Arc Post-Mortem Summary. Built the email-list subscribe and unsubscribe flow. It added a per-group mail subscription with its checkbox (Phase A), List-Unsubscribe headers on group and bulk mail plus a web unsubscribe path (Phase B), registration and whole-domain unsubscribe (Phase C), and the bot mailbox with MailSite configured (Phase D). Existing behavior was verified first and the design decisions were resolved with Chris.
Arc started 2026-06-17. This is the List-Unsubscribe piece of
v10 ToDo item 12 (Message-ID / MIME-Version / Content-Type /
Content-Transfer-Encoding and DKIM signing already landed; the
List-Unsubscribe headers plus the web + mailto unsubscribe machinery
are what remain). Goal: let a
user turn off bulk mail from a group, keep unsubscribed users out of
the recipient lists, and attach RFC 8058 List-Unsubscribe headers
(a one-click URL and a mailto address) to bulk and registration mail,
backed by a web endpoint and — when MailSite is configured
— a bot@<domain> mailbox that processes mailto
unsubscribes.
USER_GROUP (USER_ID, GROUP_ID, STATUS,
JOIN_DATE), primary key (GROUP_ID, USER_ID)
(ProfileModel fresh schema). No mail-subscription
column yet.GroupModel::getThreadFollowers($thread_id, $owner_id,
$exclude_id) returns USER_ID / USER_NAME / EMAIL for everyone
who posted in a thread plus the owner; it does not currently know
the thread's group. GroupModel::getGroupUsers($group_id,
$filter, $sorts, ...) returns every member.[subject, body, to, from] in per-batch
.txt files; BulkEmailJob reads them and
calls $mail_server->sendImmediate(...). The enqueue
points with group context are in SocialComponent near
line 1516 (thread followers) and line 1911 (group users).MimeMessage::build($from, $to, $subject, $body,
$attachments, $extra_headers) already accepts an
$extra_headers array — the natural seam for the
List-Unsubscribe headers.SmtpClient::sendImmediate wraps the built message in
self::dkimSign(...) (which calls
DkimKey::sign when a selector is configured, pass-through
otherwise), so the List-Unsubscribe headers added via
$extra_headers end up inside the signed message.RECEIVE_MAIL
column to USER_GROUP (default 1 = subscribed) in the
fresh schema, plus the portable v108 ADD ... DEFAULT 1
migration that switches it on for every existing membership.GroupModel:
getGroupUsers gained a $subscribed_only flag
(filters RECEIVE_MAIL <> 0);
getThreadFollowers gained a $mail_only flag
that, via a correlated NOT EXISTS on
thread→group→USER_GROUP, drops members with
RECEIVE_MAIL = 0. The two mail callers in
SocialComponent (post→thread-followers,
owner-post→group-users) now pass the flag; the contact-requests
caller of getThreadFollowers stays unfiltered.editgroup) and the read-only
(infogroup) views as a row in the settings table (tied
by its HTML form attribute to a separate hidden form, so
ticking it saves only this choice and the main Save leaves it alone;
a new changemailpref case checks the
viewer is a member, via checkUserGroup, then writes
their RECEIVE_MAIL and returns to the page it came
from). Driven by new GroupModel
getMailSubscription / setMailSubscription.
So a read-only viewer can still unsubscribe themselves.THREAD_UNSUBSCRIBE table (v109) recording per-user
opt-outs; GroupModel isThreadUnsubscribed /
setThreadUnsubscribe; the second arm of the
getThreadFollowers exclusion (a NOT EXISTS
on that table); and a Follow / Unfollow button on the thread view,
wired to a togglethreadmail feed action that sets the
opt-out (idempotent, so a refresh does not re-toggle) and returns to
the thread.INSERTs broke
when a column was added to a table they target. (1)
GroupModel::addUserGroup's
INSERT INTO USER_GROUP VALUES (?,?,?,?) now names its
columns so RECEIVE_MAIL defaults — this is what
made joining / creating a group fail. (2) Createdb.php fresh-install
seeds: the eight USER_GROUP seeds now supply RECEIVE_MAIL
(= 1) and the four SOCIAL_GROUPS seeds now supply
RENDER_ENGINE (the OPTIONS arc added it to the schema at
v106 but never to Createdb.php). (3) Swept every table our v100+
migrations added a column to: MAIL_ALIAS (v102, DOMAIN),
SOCIAL_GROUPS (v106), USER_GROUP (v108) — MAIL_ALIAS has no
positional inserts, so the other two were the only casualties.
Verified the fixed seeds insert cleanly at the current schema and the
old value-counts still fail. Going-forward rule: when a table is
altered, scan every insert into it.THREAD_UNSUBSCRIBE table (which
recorded "stop mailing me") is replaced by THREAD_FOLLOW
(which records "mail me about this thread"), via a v110 migration that
drops the old table and creates the new one. GroupModel
gains isThreadSubscribed /
setThreadSubscribe and a getThreadSubscribers
that lists a thread's followers with their addresses; the post-notify
path mails that list, and getThreadFollowers reverts to
its plain "thread participants" form for the contact-requests caller.
The toggle action and the (temporary) Follow / Unfollow button now use
a follow flag. Open choices: (a) icon will
be a bell (following) / bell-with-slash (not following) unless Chris
prefers another; (b) thread-follow currently mails replies independent
of the group's RECEIVE_MAIL switch — following a thread mails you
even with group mail off (alternative would be to treat RECEIVE_MAIL as
a master mute).iconlink helper. The breadcrumb bell uses
renderButton (a boxed icon link); the feed-row bell uses
renderFormButton with an onclick, exactly like the
AI-summary button beside it, so they match in size and styling. Also
tidied that helper method: fixed the additional_attibutes
parameter typo (now additional_attributes) and switched
the feed's AI-summary echos to e(). The bell
sits in the breadcrumb menu bar right after the thread name on the
thread view, and on each thread row in
the feed just after the AI-summary control. The feed needs to know the
follow state of every thread it lists, so the controller looks them up
in one query (getFollowedThreads) and the toggle action
now takes a follow_thread id so a feed row can follow the
thread it names and return to the feed.Foundation (done): the make / parse primitive that items 3 and 4
both share — UnsubscribeToken::make($user_id, $group_id)
/ parse($token) in
src/library/mail/UnsubscribeToken.php: the two ids plus an
AUTH_KEY-keyed HMAC, joined by dots, URL safe. Item 3 builds
the link with make; item 4 validates it with
parse. Covered by UnsubscribeTokenTest
(round-trip, tampered ids, forged signature, malformed input). This is
preparation, not a numbered deliverable.
MailSiteFactory::unsubscribeMailtoAddress($root_email)
returns bot@<first MAIL_DOMAIN> when this Yioop runs
its own mail, otherwise the root account address passed in. Kept it
pure (reads config, takes the root e-mail as a parameter, touches no
model) so as not to widen the factory's existing model-use exception.
(Also fixed two pre-existing >80 lines in GroupfeedElement noticed
while here.)$extra_headers
argument to SmtpClient::send() and sendQueue()
(forwarded to sendImmediate, which already supported it),
queued records now serialize a five-element tuple, and
BulkEmailJob's two readers accept four- or five-element
tuples so queued files written before the upgrade still send (with no
extra headers). Also added the missing is_array guard in
doTasks so a corrupt queued record no longer trips
count(false).MimeMessage::build's $extra_headers:
List-Unsubscribe: <mailto:...>, <https://...>
and List-Unsubscribe-Post: List-Unsubscribe=One-Click
(RFC 8058). Built by a pure
UnsubscribeToken::listUnsubscribeHeaders($user_id, $group_id,
$base_url, $mailto_address) — the web link is the item 4
endpoint with a make token, the mailto is item 1's
unsubscribeMailtoAddress carrying the same token in its
subject, plus the one-click marker. Wired into the group-users bulk
path in SocialComponent (the new-thread-by-owner
notification): each recipient's message now carries headers that turn
off that user + group's mail on the site it went out from. Covered by
a new listUnsubscribeHeadersTestCase; full suite green.
Open (Chris to decide): the other group mail
path, thread-follow notifications, is deliberately left untouched
— a group unsubscribe there would be wrong (it would
turn group mail off while the follow mail, which is independent of
RECEIVE_MAIL, kept arriving). That path wants its own
thread-level unsubscribe (drop THREAD_FOLLOW for that
thread), which is a separate token + endpoint action. The owner
join-request notice (a single-recipient operational mail) is left as
plain mail.ApiController, whose role is being widened from
the LLM activities (summarize / transcribe / translate) to general
endpoints. As prep, ApiController was brought to style
conformance this patch (wrapped four long lines, the one
// comment turned into /* */,
$result_obj renamed off the banned _obj
suffix to $decoded, generic $model renamed
to $llm_model).
✓ Done (this session): added the
unsubscribe activity to ApiController. It
cleans the token, runs it through
UnsubscribeToken::parse, and on a bad token shows a short
"invalid or expired" page. For a good token it looks up the group
name; a GET shows a confirmation page whose button POSTs the token
back (so link scanners following the GET cannot unsubscribe anyone),
and a POST calls GroupModel::setMailSubscription(..., 0)
to turn that group's mail off for that user. The page is a small
self-contained localized HTML document written straight to the
response and ended with webExit(), the same emit-and-stop
idiom the API JSON path uses, so it renders identically on the atto
server and Apache. Five api_unsubscribe_* locale strings
added. Covered by a new ApiUnsubscribeTest (the repo's
first controller test): it builds a throwaway membership row in an
isolated SQLite database, injects it into the controller, and checks
that an invalid link shows no form and changes nothing, a GET shows
the confirm form carrying the token without unsubscribing, and a POST
actually flips RECEIVE_MAIL to off. Full unit suite
40907/40907 green.
✓ Refactored (Chris's review): the
activity no longer prints HTML or branch on
$_SERVER['REQUEST_METHOD']. It now returns view data and
lets a new UnsubscribeView (web layout) draw the page
directly — no separate element, since nothing here is swapped
out — and it decides
confirm-vs-unsubscribe from a request parameter
(List-Unsubscribe=One-Click, the RFC 8058 marker the
mail client or the confirm button sends) read through
$_REQUEST, keeping the endpoint agnostic to GET vs POST.
Test reworked to assert the returned state and the flipped
subscription; full suite 40913/40913 green.
✓ Extended (this patch) for
whole-site unsubscribe: the same endpoint now also accepts the
whole-site (email-address) token. After the member-and-group parse
fails it tries UnsubscribeToken::parseEmail; a valid
one-click then calls MailSuppressionModel::suppress to
turn off all of this site's mail to that address, and
UnsubscribeView gained an all-mail confirm / done
message (two new api_unsubscribe_all_* locale strings).
Routing is by token shape, so the group path is untouched.
ApiUnsubscribeTest gains whole-site confirm and
one-click-suppress cases.dataIntegrityCheck consults the
suppression list and, when the address is on it, refuses the account
and shows the user a message that the address can no longer be used;
no account is created and no activation mail is sent. The check is
gated to registration only (password recovery stays transactional, so
an existing user can still recover). Covered by
RegisterSuppressionTest.MAIL_SUPPRESSION table (one row per
address, unique index on EMAIL) added to the fresh schema and via
upgradeDatabaseVersion111 (DATABASE_VERSION 110 →
111), plus a MailSuppressionModel with
isSuppressed / suppress /
unsuppress (case-insensitive, mirrors
MailSenderAllowModel), covered by
MailSuppressionModelTest.
Skip — the check lives in BulkEmailJob (the queue
consumer). Before sending each queued message it holds the recipient
back only when the message is list mail — it carries a
List-Unsubscribe header — and the address is on the
suppression list. Transactional mail such as a password reset carries
no List-Unsubscribe and is always sent, so a suppressed
address can still recover their account. Covered by
BulkEmailJobTest.RegisterController::sendActivationMail now adds the
whole-site List-Unsubscribe headers (one-click via the bot mailbox
and web links, from listUnsubscribeHeadersForEmail) and
a plain visible "stop all mail from this site" line at the foot of
the message pointing at the web endpoint
(UnsubscribeToken::unsubscribeUrl, shared with the
header builder so the two links never drift). Only the activation
mail carries it — password recovery stays plain. Safe because
the registration gate already refuses a suppressed address, so this
mail never goes to a suppressed recipient. The group and whole-site
header builders now share one private helper; covered by
UnsubscribeTokenTest.bot mailbox (MailSite configured)bot is reserved — it is added to the registration
forbidden-name list, so no account may be created as bot
and the bot@<domain> mailbox identity stays owned by
the system. Covered by RegisterSuppressionTest.BulkEmailJob
reads incoming bot@<domain> mail and applies the
same unsubscribe a click would. The forgery guard
(UnsubscribeToken::makeEmail / parseEmail,
earlier patch) signs the email address itself with the site secret
under a distinct all-mail prefix; the bot trusts only this
signature, never the spoofable From, so an altered
signature, a signature reused under a different address, or a plain
unsigned address are all rejected (covered by
UnsubscribeTokenTest). The reader
(processBotMailbox, this patch) runs at the top of the
name-server mail tick: when a local store has a bot
INBOX it walks each message, unfolds the Subject (the mailto links
produce "unsubscribe <token>"), and on a verified token
suppresses the address (whole-site token) or turns off the group
(member-and-group token) through applyUnsubscribeToken,
then flags and expunges every examined message so stray mail to the
bot address is cleared too. The store is reached through a new
shared MailSiteFactory::storage() and the mailbox name
lives in one UNSUBSCRIBE_MAILBOX constant. Covered by
BulkEmailJobTest (group token, whole-site token, forged
token ignored, mailbox process-and-clear, stray-mail clear).MailRecordCache did new LRUCache(...)
without importing it, so it resolved to a non-existent
...\library\mail\LRUCache and threw "class not
found" the moment codetool unit reached
DmarcCheckTest (which warms the cache). Added the
missing use; the full unit suite now runs to
completion. (The two Index test failures that then surfaced are
sandbox-only — PHP 8.3 here vs 8.5 on the dev box — and
pass on Chris's box, so left alone.)memory_limit is unlimited (-1, the common
CLI default), metricToInt("-1") is -1, so
every "usage > limit * fill_factor" memory check goes
permanently true (and the matching "<" loop guard permanently
false). That made the indexer seal a new partition almost every
document — so the on-disk archive's layout depended on
memory_limit, which it must not (archives are portable).
It also one-shotted FetchUrl's curl pump loop and made the fetcher
think memory was always low. Fixed with one helper,
memoryCeiling($fill_factor) in Utility.php (returns
PHP_INT_MAX when the limit is unlimited), applied at
every decision site: PartitionDocumentBundle, FetchUrl (x3), Fetcher,
and the legacy IndexArchiveBundle. The log-only site in
IndexDocumentBundle keeps printing the raw limit. With the fix the
two Index tests pass here under -1 too; full suite
40893/40894 (the one miss, VersionManager restoreVersion, fails on
plain HEAD too and touches none of this code — a separate
sandbox-only flake)...txt" file in the repo root when OCR is enabled):
ImageProcessor::saveTempFile built its temp name with
... . " . $file_extension" — the intended
"." separator was swallowed into a string literal, so
names came out as "<hash> . ico" with embedded
spaces (VideoProcessor's twin was already correct). That spaced path
was then handed unquoted to tesseract in
ComputerVision::recognizeText, so the shell split it and
tesseract saw output base ".", writing
"."+".txt" = "..txt" into the
working folder. Fixed the separator (now
"." . $file_extension) and hardened
the exec with escapeshellarg on all three path/lang
arguments so no future path can spray files into cwd. Added
tempFileNameTestCase (runs without tesseract) asserting
the temp name has no spaces and ends with the given extension; it
fails on the old code, passes on the new. Only reproduces where
TESSERACT is defined, which is why the PHP 8.3 sandbox never saw
it.restoreVersionTestCase failed about half the time because
getActiveVersion could skip the exact version asked for.
A version's folder is named by its timestamp turned into a string at
PHP's default float precision, which rounds the last digit; the
timestamp searched on is kept at full precision from the serialized
record. So restoring a version to its own exact time could compare as
just past that version and fall back to an older one (here the 1970
"initial" version, timestamp 1), leaving the expected file missing.
Fixed by rounding the incoming search value through the same string
form the folder names use, in both getActiveVersion and
(same latent boundary bug) getVersionsInRange. Proven:
a 60x repeat loop went from ~half failing to 60/60, the test is now a
steady [5/5], and the full unit suite is 40897/40897 all green.UnsubscribeToken
(encode into makeEmail,
decode into parseEmail); merged the two
near-identical signing methods (sign and
signDomain, same body differing only by prefix) into
one signPayload($prefix, $payload) with four callers;
and removed the duplicated "one-click → act → done, else
confirm" block in ApiController::unsubscribe by setting
the state once after a single scope branch. Suite unchanged at
40946/40946.RECEIVE_MAIL column. Thread level is supported at least
at the header level, with an "unfollow" control on the thread page
(Phase A item 4).crawlAuthHash so
the URL and the mailto suffix cannot be forged.bot@<domain> uses the domain the mail was going
out on, resolved at enqueue time.$extra_headers
through the send / queue / sweep path (see Phase B item 2).